Papers with VQA-v2 benchmark
Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research indicates that even state-of-the-art MLLMs still suffer from some straightforward visual question-answering (VQA) problems. |
| Approach: | They propose to use a model-based benchmark to investigate model laziness to identify models that err when answering simple visual questions about an image. |
| Outcome: | The proposed model laziness is found to be widespread in current MLLMs, including GPT-4o, Gemini-1.5-pro, Claude 3, LLaVA-1.5, LLva-1.6, and QWen-VL. |